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Abstract. This article describes the development, implementation, and results of an 
extensive assessment of students and student learning outcomes in an interdisciplinary, 
integrative learning community. This assessment project took a comprehensive view of 
student learning by examining specific data and direct and indirect measures of academic 
growth for each learner, from high school GPAs and perceptions about academic 
preparation upon matriculation, to the development of critical and creative thinking skills 
while participating in the first-year learning community, to student engagement levels in 
their senior year. Where applicable, data were compared to data for students who did not 
participate in the learning community. The results of this panoramic assessment project 
indicate that interdisciplinary learning and learning community practices are effective 
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in promoting academic improvement, retention, development of general education 
skills, and high levels of student engagement and can provide first-year students with an 
academic edge that follows them through their undergraduate careers. The article further 
discusses the value of using this type of 360-degree assessment to inform curricular 
decisions as well as to create institutional support for interdisciplinary, student-centered 
learning. 

Keywords: interdisciplinary learning, integrated learning, learning community, 
direct assessment, indirect assessment, retention, high impact practices, student 
engagement 


Introduction 

For decades, the value of learning communities and of interdisciplinary 
learning has been discussed and documented (Astin, 1993; Dewey, 1938; 
Jones, 1981; Boix Mansilla, 2004; Kuh, et al., 2005; Pascarella & Terenzini, 
1991, 2005; Smith, 1991; Tinto, 2000; Zhao & Kuh, 2004). More recently, 
in 2008, AAC&U’s LEAP initiative published its findings that clearly 
correlate participation in High-Impact Educational Practices (HIPs), such as 
involvement in a learning community, with increased student engagement 
and academic success, calling for “colleges and universities to make 
participating in high-impact activities a reality — and a priority — for every 
student” (Kuh, 2008, p. 22). A comprehensive programmatic review of the 
Integrated Studies Program at the University of North Dakota, a 27-year-old 
interdisciplinary learning community, supports the view that students who 
participate in such learning experiences gain an academic edge that follows 
them throughout their undergraduate experience. 

The Integrated Studies Program (ISP) at the University of North Dakota 
(UND) is a long-term learning experiment that has proven the value of 
fostering critical and creative thinking and engagement through integrative, 
interdisciplinary learning. Essentially, the Integrated Studies Program was 
created as a first-year general education learning community that offers 
students at a public research university the opportunity to take a cluster of 
four fully integrated general education courses each semester (13 credits) 
- courses in science, humanities, social science, and communications. The 
curriculum for these courses is developed and taught by a faculty team and 
offered in a student-centered learning environment where students engage 
in active learning projects, primary research, and lively discussions of 
challenging and pertinent texts. Work and study in the program are organized 
around a central theme and are carried out in a variety of small group settings 
in which discussion among students is encouraged. Pedagogically, one of 
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the most important aspects of the program is that “it consistently attempts to 
break down the barriers between disciplines and draw together the various 
disciplinary areas into comprehensive, connective units that demonstrate the 
relationships between the different fields of knowledge” (Carmichael, 2004). 
Students can complete one or both of their first-year semesters in the Integrated 
Studies Program. Understanding the core questions and approaches of each 
discipline, making connections between disciplinary issues, thinking through 
the meaning of those connections, and studying the effect of one topic on 
another, particularly as they relate to topics of personal concern, create a 
more engaged learner who develops and sustains a curiosity about the world 
and his or her place in it and is therefore prepared to be more successful 
in college and beyond (Carmichael, 2005). At the very least, integrative 
interdisciplinary learning allows students to develop a more holistic view 
of their world and to better understand the way they each can navigate in it. 
As Boix Mansilla observes, “preparing young adults to be full participants 
in contemporary society demands that we foster their capacity to draw on 
multiple sources of knowledge to build deep understanding” (2004, p. 2). 

The value of participating in common intellectual experiences, learning 
communities, collaborative assignments and projects, and global learning 
opportunities has been documented (Kuh, 2008), and the history of learning 
community pedagogy can be traced back to the colonial colleges, which 
strove to create communities of scholars with common values, to the lyceums 
of ancient Greece, and even to the ancient Jewish yeshivot of Talmudic study, 
to make the point that “community, where people join in small groups to 
discuss, explore, and learn together, has existed for centuries as a central 
concept of learning” (Lenning & Ebbers, 1999, p. 1). The development of 
contemporary learning communities can be traced back to the theories and 
experiments of educational pioneers John Dewey, Alexander Meiklejohn, 
and Joseph Tussman (Carmichael, 2004; Jones, 1981; Lenning & Ebbers, 
1999; Matthews, Smith, MacGregor, & Gabelnick, 1997; Price, 2005; 
Smith, MacGregor, Matthews, & Gabelnick, 2004; Zhao & Kuh, 2004). The 
Integrated Studies Program at UND has combined the practices of the learning 
community with an academic approach that integrates interdisciplinary 
learning, offering students both the support and connection of a community 
of learners and the academic experience of developing a perspective based on 
the integration of ideas and information from the humanities, social sciences, 
sciences, and communications. 

Assessment data we have analyzed over the years have indicated 
in isolated ways that this learning environment and approach have been 
successful, but we wanted to know more. How can this success be fully 
understood, and how can the learning outcomes for students in this program 
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be compared with those of their peers who have participated in traditional 
university learning experiences? The answers to these questions are important 
for the continued development of curriculum, assignments, and learning 
activities and for the promotion of the program’s success to administrators and 
other faculty members at the University. Thus, a multi-layered programmatic 
assessment plan was developed. This assessment plan utilizes a three-part 
approach, involving institutional data, direct student learning outcomes, and 
indirect assessments of student engagement levels to understand 

1. Who are ISP students? 

2. What do ISP students learn? 

3. What are the long-term effects of ISP participation? 

This information was analyzed and compared to the data for first-year 
students who participated in traditional stand-alone courses at UND in order 
to gain a clear picture of the effects of participation in the ISP interdisciplinary 
learning community. 

Comprehensive Learning Community Assessment: An Overview 

Each year, 60 to 80 first-year students matriculating at the University of 
North Dakota select the Integrated Studies Program and its general education 
courses instead of the traditional, non-leaming-community courses at UND. 
Any student entering UND is eligible for enrollment in ISP, where there 
is a “first come, first served” enrollment process. Approximately one-third 
of students enrolling reserve a space in the learning community ahead of 
summer orientation (during the students’ senior years in high school). They 
find out about ISP through recruitment efforts (mailings, campus visits, etc.) 
and by referral. Approximately one-quarter of enrolled ISP students choose 
ISP due to the recommendation of a friend or family member. The rest of the 
students learn about ISP during summer orientation and select enrollment at 
that time. Since the courses in ISP are primarily first- and second-year-level 
courses, no transfer students select this learning community option. 

These enrolled students form a cohort, taking a block of linked courses 
that integrate humanities, science, social science, and communications around 
a common theme. The program was modeled on and created with help of 
faculty from The Evergreen State College and is, thus, a direct descendant 
of the Meiklejohn-Tussman model (Asbeck, 1993; Carmichael, 2004). 
Primarily, it features a student-centered learning environment where the 
curriculum is created by an interdisciplinary team of faculty members who 
work as fellow-learners alongside the students. The development of skills in 
critical thinking, discussion, and collaboration is emphasized, and students 
learn to tackle difficult and complex real-world issues in a cooperative, safe 
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learning environment where trust is fostered between students and faculty. 
Faculty help to create this learning community atmosphere by taking 
students on camping trips, attending cultural events with them, and utilizing 
student discussion seminars as a main feature of the classroom experience 
(Carmichael, 2004). By exploring the differences and similarities between 
these students as entering freshmen and their peers, by studying their learning 
outcomes throughout the year, and by comparing their academic success and 
levels of engagement as they move beyond the learning community, ISP 
faculty have gained a true picture of the effectiveness of this interdisciplinary 
learning community experience. 

To gather this information, the following timeline was established: 

• Fall and Spring, 2012-13: Gather direct student learning artifacts 
from current students; 

• Spring 2013: Finalize assessment rubric; 

• Summer 2013: Gather institutional data: student demographics, 
high school GPAs, Cooperative Institutional Research Program 
(CIRP) data, National Survey of Student Engagement (NSSE) data; 

• Fall 2013: Norm assessment rubric (faculty); 

• Fall 2013: Apply rubric to student learning artifacts and analyze data. 

The assessment director gathered the information and led the norming 

sessions (described later). Data analysis was provided both by the assessment 
director and by UND’s Office of Institutional Research. (Institutional Review 
Board permission for this study had been obtained.) 

Institutional data were collected from the past five years to provide the 
most up-to-date portrait of the current ISP student and give the most accurate 
information for comparison with non-ISP students. Additionally, analyzing 
data over a five-year period, where appropriate, provided a sense of continuity 
or trends in characteristics. 

The data collected and analyzed were applied to address the three 
questions guiding the program review to better understand who ISP students 
are, what ISP students learn, and what the long-term effects of ISP on student 
learning and engagement are. Since several key campus-wide surveys were 
found to be essential to this study (especially the NSSE), it was determined 
that this study would take place every three years in concert with the collection 
of data from those instruments. 

Who Are Integrated Studies Program Students? 

To understand the impact of participation in this interdisciplinary 
learning community, first the similarities and differences between the 
students and their peers at UND must be understood. To determine who ISP 


58 


Tami Carmichael & Yvette LaPierre 


students are when entering ISP, five years’ worth of institutional data were 
analyzed, specifically information on gender, hometown, high school grade 
point average, and self-reported CIRP data that provide metrics on students’ 
high school experiences and academic expectations in college. Some 
information from these data is also important for determining the level of 
academic success that can be expected from students in their undergraduate 
career (Pasque & Murphy, 2005; Shapiro & Levin, 1999; Smith, 1991). 

After faculty analyzed and compared the data, the following portrait of 
ISP students emerged: 

• Demographics: Like their peers at the University of North Dakota, 
incoming first-year students in the Integrated Studies Program over 
the past five years tend to be approximately 18 years old, from the 
same mix of public and private institutions, primarily from North 
Dakota and Minnesota, and of the same gender mix (slightly more 
heavily female than male). However, unlike their peers’ grade point 
averages, ISP students’ high school grade point averages tend to be 
lower, with ISP students’ average at 3.30 on a 4-point scale and non- 
ISP students’ average at 3.38. 

• Perceptions of Learning/Academic Preparation: According to 
CIRP data gathered from surveys completed by incoming first-year 
students and most recently processed at UND in 2011, ISP students 
demonstrate the following in comparison to their peers also entering 
UND in that same year. Thirty-five ISP students were surveyed. 
Statistical significance was determined using a /-test. Indicators 
were determined significantly different at p <.01 : 


CIRP Metrics 

ISP Students 

Non-ISP 

Students 

Completion of College-Prep Math 
Courses 

Pre-Calc: 48%* 
Calculus: 0%* 

AP Calc: 5%* 

Pre-Calc: 75% 
Calculus: 23% 

AP Calc: 18% 

Self-Perceptions of Math Abilities 
(Likert Scale where 1= Not able, 
4=Very able) 

2.6* 

3.5 

Self-Perceptions of Academic 

Abilities 

(Likert Scale where 1= Not able, 
4=Very able) 

3.2* 

3.8 
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Plans to Obtain Bachelor’s Degree 
(Likert Scale: l=No chance, 4=Very 
good chance) 

19%* 

42% 

Plans to Obtain Graduate Degree 
(Likert Scale: l=No chance, 4=Very 
good chance) 

9%* 

31% 

Important Personal Goal: Develop 
Meaningful Philosophy of Life 
(Likert Scale where l=Not important, 
4=Essential) 

2.5* 

2.0 

Important Personal Goal: Influencing 
Social Value 

(Likert Scale where l=Not important, 
4=Essential) 

2.5* 

2.0 

Important Personal Goal: Study 

Abroad 

(Likert Scale where l=Not important, 
4=Essential) 

2.8* 

2.0 

Future Plans: Plans to Change Major 
(Likert Scale: l=No chance, 4=Very 
good chance) 

2.8* 

2.4 

Future Plans: Plans to Change Career 
Choice 

(Likert Scale: l=No chance, 4=Very 
good chance) 

3.0* 

2.4 


* Demonstrates Statistically Significant Difference 


Discussion and Limitations: 

Overall, incoming ISP students match the same age, gender, and regional 
demographics as non-ISP students at UND, but they tend to have lower high 
school grade point averages than non-ISP students, are less well-prepared for 
college-level courses, and have a lower sense of academic ability, particularly 
in mathematics. A noteworthy difference indicated in these data is the great 
disparity in ISP students’ self-reported plans to complete a Bachelor’s degree 
(only 19%) compared to their non-ISP peers’ belief that they would complete 
that degree (42%). This information would seem to indicate that students 
enrolling in ISP have a less clear vision of their academic goals and less 
confidence in their abilities to complete college or both. Thus ISP may 
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be serving a population of students who may be at great risk of not being 
retained by the institution but who, studies report, are especially well-served 
by participating in high-impact practices, where the “salutary effects are even 
greater for students who begin college at lower achievement levels” (Kuh, 
2008, p. 19). 

As in the data from any self-reported survey, the degree of certainty 
over outcomes is somewhat compromised by user perceptions and 
misunderstandings. However, the overall sample size of ISP and non-ISP 
students was robust, which should correct for many of these inaccuracies. 
Participation in the CIRP survey is voluntary, so that data would be affected 
by outside influences on students’ choices to complete the survey; however, 
50 percent of the incoming ISP cohort did complete this survey. Additionally, 
some responses to questions, such as the one addressing students’ goals for 
completing a Bachelor’s degree, might be compromised as some students 
may not understand the term “Bachelor’s degree.” However, many of these 
data are simply reported, factual data (such as high school GPA), which 
would be more reliable. 

What Do Integrated Studies Students Learn? 

Integrated learning, of which interdisciplinarity is a subset (Larder 
& Malnarich, 2009, p. 32), “posits that truth is not the act of ultimately 
establishing knowledge, but rather the activity of its provisional progress” 
(Welch, 2011, p. 3). By being exposed to the ways that different disciplines 
consider an issue and by learning to compare the different approaches of 
those disciplines, students can begin to form a more complete and meaningful 
perspective and make more informed decisions. Additionally, integrating 
ideas from multiple disciplines helps students create a context for their own 
learning. Integrating ideas in an interdisciplinary learning environment more 
accurately reproduces a real world environment where individuals take in 
various pieces of information from interdisciplinary sources and combine or 
integrate them in different ways to reach conclusions. 

To guide students through this activity of integrating information and 
ideas from multiple disciplines, faculty members introduce students in the 
University of North Dakota’s Integrated Studies Program to four specific 
learning goals: 

1. To learn to compare ideas to consider alternative ways of thinking; 

2. To synthesize and make connections between ideas and topics and 
to advance new ideas as a result; 

3. To analyze topics in-depth and use findings to formulate new 
concepts or reach new conclusions; 
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4. To connect and integrate ideas across disciplines in order to posit 
new or alternative ideas or explanations. 

The careful development of these goals by ISP faculty was guided 
by ongoing feedback gathered from the assessment process developed by 
Boix Mansilla for evaluating integrative work (Boix Mansilla, 2004, 2007). 
Assignments incorporate these goals, and student work is assessed according 
to them. It is essential to any assessment experience to establish and articulate 
to students clear learning goals against which they are continually evaluated. 

To make learning gains in these areas, students participate over the 
semester in a series of readings, intentionally integrative assignments, and 
intense discussions, where they are asked to consider the way knowledge 
is acquired in the disciplinary areas of humanities, social science, science, 
and communications. They are asked to consider and interact with primary 
and secondary texts or materials in these disciplines in order to practice and 
develop critical and creative thinking, and then they are asked to bring the 
information from those disciplines together and to consider where ideas in 
each area brush up naturally, and sometimes even not so naturally, against 
each other. They are asked to compare and contrast these ideas and to discuss 
the ways in which the various pieces of information support or contradict 
the other pieces. Finally, they are asked to consider new ideas or information 
that may come to light after they have connected, compared, and contrasted 
these interdisciplinary perspectives and to work toward new conclusions that 
emerge in the process of these considerations. This work of integration, of 
combining or connecting topics and ideas across disciplines and applying 
conclusions drawn from that work to new situations, allows students to see 
the natural progression of idea development. It becomes an “effective strategy 
for comprehending, navigating, and transforming knowledge” (Welch, 2011, 
P-2). 

Work in ISP classes involves a great deal of oral processing through 
seminar discussions and oral exams. Students are expected to hone their 
intellectual skills by responding to and discussing integrative questions and 
topics within a learning community setting. Since the goal of assessment 
is to provide an authentic look at students’ direct learning process, faculty 
members decided to analyze that learning in its organic setting. The 
seminar style of ISP classes, it was felt, provides the best setting for this 
data collection. Therefore, to monitor students’ learning, a series of authentic 
assessment activities have been put in place throughout each semester. Called 
“checkpoints,” these experiences require that students read and prepare a 
significant document or book and participate in an intense oral discussion 
of that work with three fellow students for 40 minutes without faculty 
participation until the end of the checkpoint exam. For instance, after study 
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and discussion of the U.S. Constitution , articles on genetics and genetic 
testing, and philosophical debates on what it means to be human, students 
might be assigned a text like The Immortal Life of Henrietta Lacks (Skloot, 
2010). It would be their job to read and prepare this text for discussion in light 
of their recent work in the three disciplinary areas of social science, science, 
and humanities (philosophy). 

Four of these checkpoint experiences are scheduled over the course of 
the semester. Students self select into groups of four a week ahead of the 
checkpoint and are given the text they will be responsible for preparing and 
discussing. They are encouraged to meet together ahead of time to study and 
discuss the new work, and they are given copies of the rubric that will be used 
to evaluate them. Of course, they have also been presented with the overall 
learning goals of the program. During the checkpoint, students engage in a 
discussion of the work without feedback, input, or assistance from faculty. 
They have experienced seminar discussions with their classmates in groups 
of 10 to 20, with faculty interaction and facilitation. However, in a checkpoint 
experience, the discussion becomes much more intense and is guided entirely 
by the students in the group. The level of the preparation and of their previous 
engagement in the work they have done throughout the semester in ISP will 
have a clear effect on their performance in this situation. In the final 10 to 
15 minutes of the checkpoint, faculty will select an idea or topic that the 
students have brought forward and ask them to go back to that topic and 
discuss it in more detail. This provides an opportunity for groups to dig more 
deeply into an important idea they may have disregarded too quickly. This 
also gives faculty a chance to ask group members to address important ideas 
they have missed entirely. For instance, they may have shaped a discussion 
that integrates social science and humanities but doesn’t include science. A 
faculty-directed question at the end gives them a chance to try to do this. 
They may fail or they may succeed, but either way the faculty member gets a 
sense of the level of their integration ability. 

Faculty monitor the checkpoint and record the types of student responses 
according to the following rubric, developed over several years with some 
guidance from the Association of American Colleges and Universities’ 
guidelines for assessing critical thinking and integration (AAC&U, 2010). 
This rubric also contains assessment points for topics other than critical 
and creative thinking and integration, providing faculty and students with 
information on a student’s progress in other key areas for academic success 
including text preparation, listening, and overall participation: 
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Student Evaluation/Checkpoint Rubric, Integrated Studies 
Program 


Basic Skills 

Level 1 

Inactive 

Emerging 

Level 2 
Exploratory 

Emerging 

Level 3 

Integrative/ 

Insightful 

Preparing 

Materials 

Little evidence 
of text 
preparation 


Evidence of 
text preparation 


Evidence of 
more than just 
text preparation 

Participation 

Little 

participation 


General 
observations 
on assigned 
materials 


Active/specific 
participation 
using assigned 
texts and other 
materials 

Personal 

Relevance 

No evidence 
of connecting 
materials 
to personal 
experiences 


Makes multiple 
connections 
to personal 
experiences/ 
may not 
advance 
discussion or be 
relevant 


Connections 
between 
material and 
personal 
experiences 
are relevant 
and advance 
discussion/ 
analysis 

Using Texts/ 
Materials 

Few references 
to assigned 
text or course 
materials 


Multiple, 
general 
references to 
assigned text 
and course 
materials 


Multiple/ 
significant 
specific 
references to 
assigned text 
and course 
materials 

Listening 

Distracted/ 

unengaged 


Listens to 
others, may 
respond 


Listens and 
responds 
to others in 
meaningful 
ways 
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Comparing 

Ideas 

Few 

comparisons 

made 


Some general/ 
evident 
comparisons 
made between 
ideas, texts, 
experiences 


Multiple 
comparisons 
made between 
texts, ideas, 
experiences; 
some may verge 
on insightful 







Advanced 

Skills 

Level 1 

Emerging 

Level 2 

Emerging 

Level 3 

Connecting/ 

Integrating 

No connections 

across 

disciplines 


Occasional, 

observational 

connections 

across 

disciplines; may 
notice contrasts 


Frequent and/ 
or thoughtful 
connections 

across 
disciplines; 
notes contrasts 

Extending 

Discussions 

Little extension 
of others’ 
comments 


Begins to 
explore others’ 
comments; 
does more than 
agree/disagree 


Considers 

others’ 

comments 

and works to 
integrate those 
ideas with 
other discussed 
issues/texts 

Synthesizing/ 

Projecting 

Does not 
attempt to 
synthesize ideas 


Explores 
conflicting info/ 
ideas and works 
to synthesize 
differences 


Synthesizes 
conflicting 
info/ ideas and 
introduces new 
topic as a result 

Analyzing/ 

Applying 

Little or no 
exploration of 
topic or text 
or drawing 
of larger 
conclusions 


General/ 
surface 
exploration 
of topic/ text; 
begins to 
reach larger 
conclusions 


Explores topics 
in-depth and 
uses findings 
to draw 
conclusions/ 
formulate new 
insights 


(Carmichael, LaPierre, Magness, 2012). 


Responses on this rubric are recorded and transferred to a Likert scale 
where l=Inactive, 2=Emerging to Exploratory, 3=Exploratory, 4=Emerging 
to Integrative/Insightful, and 5=Integrative/Insightful. All faculty members 
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using this rubric underwent norming sessions to assure inter-rater 
reliability using two student checkpoint seminars that were videotaped with 
permission of the student groups. All five faculty who assessed checkpoint 
experiences with the rubric watched the videos, recorded their assessments, 
and compared the results. A discussion of any discrepancies followed and 
all were resolved. In addition, an outside expert in this style of learning also 
watched and assessed the videos. Comparisons with the group results were 
made, and the instrument was found to be reliable. 

Following these norming sessions, each faculty member applied the 
rubric to four sets of his or her own checkpoints — two from fall semester 
checkpoints and two from spring semester checkpoints. In both cases, the 
first and final checkpoints were used to determine significant differences in 
the amount of learning across a semester. Random samples of rubrics from 
all faculty were selected for analysis and results were averaged. Assessments 
for the same students were used from both the first and final checkpoint 
assessments in each semester. Since this was the pilot test of the instrument, 
only 12 samples were selected each semester. In further assessment work, 
the sample selection will be increased to at least 20. Random selection was 
carried out by an outside staff member. The student cohort assessed here is 
the same cohort for whom demographic and CIRP data were collected and 
analyzed as described in the previous section. Students can complete one or 
both semesters of their first year by enrolling in the integrated block of 13 
ISP credits each semester; therefore, in order to assess only students who had 
been enrolled in ISP courses for their entire first year, in the second semester 
only data from returning students were used. 1 A paired /-test two-tailed Fall 
1 vs. Fall 4 was conducted to determine significance of difference. This is 
the first year that a complete set of data has been collected and assessed. 
In subsequent years, faculty members will need to compare data across the 
years to see what trends emerge. This is the intention of the assessment plan. 


1 Because many students who start in the fall choose to pursue other academic oppor¬ 
tunities at UND in the spring semester (such as beginning courses for their major), 
a smaller group of students enters the spring semester in ISP (approximately two- 
thirds of the original fall coh ort). Students choose to leave ISP after one semester for 
varying reasons each year; currently more than 80 percent of those leaving report on 
an exit survey that they are not continuing due to the need to “take required courses 
in the major.” This attrition could result in lower or higher second semester averages 
depending on the academic strength of students continuing. 
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Following are the results for the assessment areas pertaining to critical 
and creative thinking and interdisciplinary integration: 



Comparing 

Ideas 

Synthesizing/ 

Projecting 

Analyzing/ 

Applying 

Connecting/ 

Integrating 

Semester 1: 

First Checkpoint 
(September) 

2.0 

1.25 

1.25 

1.25 

Final 

Checkpoint 

(December) 

3.92* 

3.88* 

4.29* 

4.17* 

Semester 2: 

First Checkpoint 
(February) 

4.41 

3.87 

4.24 

4.17 

Final 

Checkpoint 

(May) 

4.42 

3.96 

4.25 

4.17 


Semester 1, n=12; p < 0.05 Semester 2, n=12; 

Likert scale: l=Inactive, 2=Emerging to Exploratory, 3=Exploratory, 
4=Emerging to Integrative/Insightful and 5=Integrative/Insightful. 

*Fall Checkpoint results demonstrate significant difference. 

Discussion and Limitations: 

Checkpoint experiences begin after the first quarter of the fall semester; 
therefore, students are being assessed after they have gained experience in 
reading and discussing texts and ideas at the college level. They have also 
studied various theories on logic and on logical fallacies and have watched 
other academics debate or discuss ideas in the classroom or on the Internet, via 
forums like TED talks. Additionally, they have been exposed to the ways of 
thinking in four broad disciplinary areas: humanities, social science, science, 
and communications. Thus, assessment begins after students have begun to 
gain some experience in the areas in which they are being assessed. It is hoped 
that, to some degree, this creates a more level playing field between students 
who come in with varying degrees of experience in reading, discussion, and 
analysis. Additionally, it gives the learning community time to form, creating 
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that safe space for discussion that is essential to successful student participation 
(Light, 2001; Ullah & Wilson, 2007; Yazedjian, et al., 2008). Since these data 
sets indicate learning outcomes over only two semesters of students’ first 
year of college, the expectation is not that they should be achieving a level of 
mastery (4-5) but that they should show growth in most areas over time. 

Direct assessment of student learning seems to indicate that students, 
overall, gain significantly in all areas of thinking and integration development, 
with the greatest gains occurring between the first and final checkpoint in 
the fall semester. Those gains are maintained and slightly increased by the 
final checkpoint in the spring semester, though not significantly. These 
findings corroborate the value of creating learning opportunities through 
interdisciplinary education (Boix Mansilla, 2004; Field & Stowe, 2002; 
Newell, 1998). It does appear that students are naturally more able to 
compare ideas than to integrate and analyze ideas across disciplines, which 
is expected since integration involves comparison, synthesis, and extension 
of ideas, making it more complex and difficult to master. Most students 
would have experience in high school classes with idea comparison and 
contrast but probably would not have been required to compare information 
across different disciplines in order to see patterns or reach new conclusions. 
Unfortunately, too often at the college level, opportunities for cross- 
disciplinary comparisons, and thus eventually integrative thinking, are not 
provided to students. The Integrated Studies Program at UND does provide 
such opportunities and, in fact, challenges students to be integrative in their 
thinking and reasoning, and the learning outcomes reflect the educational 
benefit of this experience. Synthesizing and integrating both remain the 
categories of lowest growth, but since these two areas require the greatest 
sophistication in abstract thinking, it is not unexpected to see lower scores 
for first year students despite their exposure to this type of thinking and 
learning. 

It is noted that there is not a significant difference in learning outcomes 
from checkpoint one to the final checkpoint in the spring semester, though 
levels are maintained. This finding has been recognized by the faculty 
team and is under discussion. It may be that students are operating at their 
capacities for their developmental level, or it may be that they are not being 
challenged in the second semester in ways that could give them opportunities 
to further develop their thinking and integration skills. As stated, this pilot 
study is only in its first year. Continuing to assess students in the same way 
and to compare data across academic years will help yield more accurate 
and informative results and will help faculty determine how best to shape 
the learning practices in the spring semester. 
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What Are Some Long-Term Effects of Participation in the 
Integrated Studies Program? 

As all of the previous studies referenced indicate, data for multiple 
decades have pointed to the benefits of student involvement in both interdis¬ 
ciplinary learning and high-impact practices like learning communities. But 
what, specifically, can be said about the long-term effects of participating in 
an interdisciplinary learning community as ISP students do versus their non- 
ISP peers? Discovering this information becomes essential as programs like 
ISP must increasingly argue for their institutional value as they vie for needed 
internal and external funding (Carmichael, 2005; Dickeson & Ikenberry, 
2010; Field & Stowe, 2002; Kuh, 2008; Vescio, 2008). Additionally, and 
most importantly, discovering the long-term effects of participation in ISP 
will help that department better understand the strengths and weaknesses of 
the curriculum, pedagogy, and learning community practices and to adjust 
them as need be. 

To determine the long-term results of ISP participation, three sets of 
external data were collected and analyzed: 

• First semester GPAs (2008-2012); 

• Fall-to-fall retention rates (2008-2011); 

• National Survey of Student Engagement (NSSE) data for first- 
and fourth-year students (2009, 2011). 

These particular metrics were chosen because of the strong evidence 
that these data sets can articulate correlations between outcomes and 
academic success. First semester GPAs have long been found to give the 
first indication of continued success in college (Krumrei-Mancuso, et ah, 
2013; McPherson & Schapiro, 2008). Retention rates are important to the 
value of a department or program as it makes arguments for its validity and 
for continued funding. And, of course, students can only be successful at 
an institution if they are retained and continue that education. The National 
Survey of Student Engagement (NSSE) is the best known national project 
for measuring the extent to which students engage in practices that have 
been linked with positive learning outcomes, personal development, student 
satisfaction, and academic persistence. The recent work of many scholars 
(Astin, 1993; Pascarella & Terenzini , 1991, 2005; Kuh, Cruse, Shoup, 
Kinzie, & Gonyea, 2008) indicates that students who are actively engaged 
in academic and co-curricular activities gain more from their college 
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experience and are more academically successful than students who are not 
as engaged. 

First Semester GPAs 2008-2012 Average 

Grade point averages were averaged over five years for both ISP 
and non-ISP students and Welch’s /-test was used to determine significant 
difference. This analysis indicates that at the end of the first semester, 
despite the lower entering high school GPAs, ISP students had a statistically 
significantly higher GPA. 


Group 

N 

Mean GPA 

Std. Dev. 

ISP 

362 

3.08* 

0.813 

Non ISP 

9,979 

2.80 

0.957 


* = significant difference, p<0 .05 

Fall-to-Fall Retention 2008-2011 


Retention rates from first to second year were factored over six years 
and show little significant difference between ISP and non-ISP students. This 
finding is important to ISP, however, when it is viewed through the original 
data indicating that only 19 percent of entering ISP students believed they 
would continue on in college to finish a Bachelor’s degree, compared to 42 
oercent of their non-ISP peers who believed they would finish that degree. 


Year 

ISP Retention Rate 

Non-ISP Retention Rate 

2006 

81% 

75% 

2007 

74% 

78% 

2008 

77% 

77% 

2009 

73% 

79% 

2010 

75% 

77% 

2011 

79% 

74% 

Average Retention Rates 

76.5% 

76.6% 
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NSSE Benchmarks 2009 and 2011 

The NSSE establishes five benchmarks of desirable educational 
practices, based on Astin’s (1984) theory of student involvement and 
Chickering and Gamson’s (1987) work, and surveys ask students to respond 
to sets of questions intended to provide insight into these benchmark areas. 
The five benchmarks are level of academic challenge (LAC), active and 
collaborative learning (ACL), student-faculty interaction (SFI), enriching 
educational experiences (EEE), and supportive campus environments (SCE). 
First-year and senior students at UND were anonymously surveyed. The data 
for ISP students were pulled out and compared with those for the non-ISP 
students. In order to determine whether ISP students reported higher levels 
of engagement than their non-ISP peers, an independent samples /-test was 
conducted to compare the means of the two samples of students for significant 
differences between the two groups of students. Results were analyzed for a 
pattern that could help explain any significant differences found. 


First-Year Students 2009 


Benchmark 

ISP Mean 
(bold is >non ISP) 

Non-ISP Mean 

LAC 

63.1* 

51.5 

ACL 

55.1* 

39.4 

SFI 

38.2 

31.0 

EEE 

31.3* 

22.8 

SCE 

64.4 

60.9 


ISP n=14 non-ISP n=514; p <.05 raw scores, non-weighted 
*= significant difference determined 


First-Year Students 2011 


Benchmark 

ISP Mean 
(bold is >non ISP) 

Non-ISP Mean 

LAC 

61.5* 

52.4 

ACL 

53.0* 

40.8 

SFI 

37.4 

32.8 

EEE 

26.4 

24.8 
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SCE 


62.6 


61.5 


ISP n= 23 non-ISP n= 525; p <.05 raw scores, non-weighted 
*= significant difference determined 


Senior Students 2009 


Benchmark 

ISP Mean 
(bold is >non ISP) 

Non-ISP Mean 

LAC 

61.3 

55.6 

ACL 

51.0 

50.1 

SFI 

52.5 

45.7 

EEE 

52.5* 

37.2 

SCE 

60.5 

55.6 


ISP n=10 non-ISP n=415;/K.05 raw scores, non-weighted 
*= significant difference determined 


Senior Students 2011 


Benchmark 

ISP Mean 
(bold is >non ISP) 

Non-ISP Mean 

LAC 

65.0* 

55.0 

ACL 

57.2 

47.3 

SFI 

60.0* 

39.8 

EEE 

58.0* 

35.8 

SCE 

59.9 

57.2 


ISP n=12 non-ISP n=783;/?<.05; raw scores, non-weighted 
* = significant difference determined 


Discussion and Limitations: 

First-year students who participated in the Integrated Studies Program 
learning community scored higher than their non-ISP peers on all five 
benchmarks of the NSSE in both first and senior years in both the 2009 
and the 2011 survey. This persistence of ISP students out-scoring non- 
ISP students indicates a consistent pattern of increased engagement over 
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multiple years and over the academic journey of students from first to senior 
years. The mean differences were found to be statistically significant for 
LAC, ACL, and EEE in the 2009 cohort and for LAC and ACL in 2011. 
In the senior year group, mean differences were found to be statistically 
significant for EEE in 2009 and for SFI and EEE in 2011. 

These studies seem to corroborate previous research on the positive 
effects of participating in learning communities and on the positive impact 
of interdisciplinary curriculum (Newell, 2008). The Integrated Studies 
Program, like most learning communities discussed in the literature, 
incorporates active and collaborative learning activities, critical and creative 
thinking skills development, high levels of student-faculty interactions, and 
continuous exposure to interdisciplinary thinking and collaboration. These 
educational activities, which are all linked to higher levels of engagement, 
are reflected in the higher NSSE scores of ISP students. And though NSSE 
does not directly assess learning outcomes, Umback and Wawrzynski argue, 
based on their review of the literature, that “if educational practices lead 
to student engagement and student engagement leads to certain outcomes 
of college (e.g., student learning and retention) then it can be said that 
educational practices indirectly lead to student outcomes from higher 
education” (2004, p.156). Certainly the higher levels of engagement 
reported here are consistent with the on-average significantly higher GPAs 
reported and the higher-than expected rates of retention from first to second 
year. Combining the results of direct assessment of student learning with 
indirect student self-assessment of learning allows for a clearer picture of 
the effectiveness of an interdisciplinary learning program like Integrated 
Studies. Being able to corroborate faculty observations and student self- 
reported data with direct learning outcomes greatly strengthens arguments 
for the value of these kinds of high impact practices. 

Plans for Further Study and Development 

Though ISP faculty have conducted meticulous and ongoing 
assessments of student learning over the 27 years the Program has existed, 
this is the first time a complete 360-degree assessment of students and their 
learning outcomes has been conducted. In some sense, then, this assessment 
is both a pilot and a baseline that requires further similar assessment so that 
trends over time can be studied. ISP faculty intend to collect direct learning 
data each semester as described in the section on checkpoints. Increasing 
the sample sizes, now that the validity of the rubric and raters has been 
established, will provide more robust results that can be used to make yearly 
curriculum decisions. These data will also provide yearly comparisons so that 
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faculty can see if and when a group seems to be below or above the recorded 
averages. Already, other departments and programs on campus are looking 
to this new model established for ISP to create similar ways of studying and 
documenting their student populations, their learning outcomes, and their 
ongoing academic achievements. 

The demographic information and CIRP data for ISP vs. non-ISP 
students will be collected annually to provide an immediate sketch of 
who entering students are and what their strengths and limitations might 
be. NSSE data will be analyzed each time they are available, and in these 
“NSSE years,” complete reports, like this one, will be compiled. Thus, every 
two years, a complete 360-degree review of students and of student learning 
outcomes will be articulated. 

Providing long-term records of this kind of information will help less- 
traditional units that emphasize interdisciplinarity, competencies-based 
outcomes, and student-centered teaching and learning articulate their value 
to the more traditional, research-based college or university and should 
allow these types of units to vie for resources and for prominence in any 
prioritization processes. 

Conclusion 

Recent major criticism of undergraduate education in the United States 
claims that “colleges and universities have failed to focus on undergraduate 
education and student learning in particular” (Ullah & Wilson, 2007, p. 
174). A recent nation-wide study of the undergraduate education system 
proclaims, “four-year colleges and universities and students attending them 
are too often ‘academically adrift,’” reflecting a sharp decline in academic 
work effort and learning from earlier decades (Arum & Roksa, 2011, p. 
30). More specifically, the authors of the study found that gains in critical 
thinking, complex reasoning, and writing skills were either exceedingly 
small or statistically nonexistent for many students, and 36 percent of 
students experienced no significant improvement in learning over four 
years of college (Arum & Roksa, 2011). Back in 1987, Chickering and 
Gamson characterized undergraduate education as a “spectator sport” (p. 5). 
According to Tinto (2000), the college experience for many U.S. students 
still remains very much the “spectator sport” Chickering and Gamson 
described it as being in which “students do not learn much just by sitting 
in classes listening to teachers, memorizing pre-packaged assignments, and 
spitting out answers” (p. 5). In his analysis of nation-wide NSSE data, Kuh 
(2003) reported that about one-fifth of both first-year students and seniors 
“frequently” come to class unprepared and say their institutions place 
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little emphasis on studying and spending time on academic work. These 
disengaged students put very little effort into their studies and report making 
very little progress toward desired outcomes in college (p. 27). 

Though there is not a single prescription for the ills that beset 
undergraduate education and its disengaged students, it is argued that “there is 
growing evidence that—when well done—a handful of selected programs and 
activities appear to engage participants at levels that boost their performance 
across a variety of education activities and desired outcomes” (NSSE Annual 
Report, 2007, p. 7). Two of these select programs or methods are learning 
communities and interdisciplinary learning (Zhao & Kuh, 2004). However, 
thus far much of the evidence that supports the belief that these methods are 
effective has been qualitative and anecdotal. Now, the information from this 
extensive, panoramic study of students in UND’s interdisciplinary learning 
community provides a solid foundation of support for these claims, based 
on quantifiable data gathered in a rigorous and reliable assessment process 
that measured student learning outcomes and engagement levels over time. 
The results clearly indicate that interdisciplinary learning and learning 
community practices are effective in promoting academic improvement, 
retention, development of general education skills, and high levels of student 
engagement. 

Beyond that, creating a comprehensive assessment process that takes 
into account student data from the beginning of their college experience, 
through their participation in the interdisciplinary learning community of 
ISP, beyond ISP, and into their senior year, allows a more developed portrait 
to be painted of the students who pursue this interdisciplinary learning 
community experience and their peers who do not. It helps the faculty of 
such a learning community to better understand whom they serve, so that 
learning can be tailored to fit the needs of the students, and it helps those 
faculty members to argue to a larger community for the value of the learning 
experience they provide. Comparing ISP and non-ISP students in terms of 
direct assessment of student learning and indirect assessment of students’ 
perceptions of learning and engagement allows faculty to answer confidently 
the question of whether participating in this type of interdisciplinary 
learning community has long-term major impacts. And it can suggest how 
non-ISP courses might take advantage of successful ISP practices. Creating 
such an extensive analysis of student learning, including both direct and 
indirect assessment, can provide the faculty of any interdisciplinary learning 
community with a clear picture of the positive effects of integrative, student- 
centered learning and will provide that unit with the tools it needs to move 
successfully into the future and to positively impact student learning in new 
and more comprehensive ways. 
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